www.rspsciencehub.com 





OSD ov VOS, 


INTERNATIONAL RESEARCH JOURNAL ON 
ADVANCED SCIENCE HUB 


RSP SCIENCE HUB 


(The Hub of Research Ideas) 
Available online at www.rspsciencehub.com 


Volume 03 Issue 05S May 2021 


e-ISSN : 2582 - 4376 
Open Access 


Special Issue of First International Conference on Information Technology, Computing & 


Applications ICITCA 2021) 


EEG based Emotion Recognition and Classification: a Review 


Ramprasad Kumawat', Manish Jain? 


"Ph.D Scholar, Department of Electrical and Electronics Engineering, Mandsaur Univesity, Mandsaur, 


Madhya Pradesh, India. 


"Associate Professor, Department of Electrical and Electronics Engineering, Mandsaur Univesity, 


Mandsaur, Madhya Pradesh, India. 
ad 
rp.kumawat@ meu.edu.in 





Abstract 





Emotion plays a vital role in medical research and interpersonal communication. Essentially feeling can 
be communicated verbally like discourse or non-verbally like outward appearance and physiological 
signals. A human emotion is complex physiological state which involves a physiological response, a 
person’s experience and behavioral change. EEG measures electric current that are generated due to 
neuronal activities in the human brain. This paper provides an overview of comparative study of various 
techniques of emotion recognition from EEG signals. Our analysis is based on extracted features and 
classification methods of emotion recognition. We intended that, this study will be useful for newly 
researchers those entering in the field of emotion recognition. 
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1. Introduction 

Essentially feeling can be communicated verbally 
like discourse or non-verbally like outward 
appearance and physiological signals. 
Physiological signals are bio electrical signals that 
are control by the automatic nerves system [1, 2]. 
The human computer interaction system (HCI) 
find lots of applications in biomedical engineering, 
neuromarketing, neuroscience and other areas of 
human life, which are affected by emotions. 
Because of increasing demand of HCI and 
automatic human emotion recognition. Currently 
emotional research focuses more in the diagnosis 
of depression, mental illness, and assists health 
care professionals to make accurate diagnosis [3- 
5]. Human emotion can be recognize through 
physiological signal like Electromyography 
(EMG), Electrocardiography(ECG), Gelvonic skin 
response and Electroencephalography(EEG) from 
all of these, EEG is more effective because it 
provide more accurate, non-invasive and 
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convenient way of capturing brain signal. And also 
EEG is portable device [6, 9]. Diagnosis of EEG 
signal varies with expertise. EEG recording require 
lot of time for manual inspection. And some time 
results may not be accurate because of artifact in 
the signals. So the processing an analysis of EEG 
signals can be done with the help of HCI 
technologies to get fast and accurate results. Using 
these technologies diagnosis of neuropsychiatric 
and neurological disorder such as Alzheimer, 
epilepsy and depressive disorder will be done [31]. 
Nowadays, researcher has shows extensive interest 
to make human emotion interaction with machine 
by proper brain computer interface [23].Literature 
survey covered recent methods used _ in 
physiological signal based emotion recognition; it 
will help for researcher working in this field. The 
remaining portion of this paper is organized as 
follows: Emotion is described in Section II. 
Characteristics and various rhythms of EEG 
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signals is describes in section III. Section IV 
describes the various databases used and _ pre- 
processing of signals. Extracted features and 
various classification techniques have been 
describe in section Conclusion and future direction 
gives in section 
2. Emotion 
A human emotion is complex physiological state 
which involves a_ physiological response, a 
person’s experience and behavioral change. An 
emotion is necessary component of human and 
significantly affects human's day by day activities 
including interaction, communication and learning 
etc.There are two _ perspectives towards 
representation of emotion. The first category 
indicate basic emotions through natural selection. 
There are eight basic emotions; fear, anger, 
disgust, sadness, curiosity, surprise, joy and 
acceptance. In the second category emotions are 
mapped into the Arousal, Valence and 
dominance[18]. Wast research on techniques of 
emotion recognition have been done over the past 
years and classified in to three catagories: 
e Periphery physiological signals 
e Facial expressions and voice 
e Brain signal generated by central nervous 
system. 
Some people cannot truly reflect their emotions 
into their facial expressions, because of some 
special reasons and a patient who suffers facial 
paralysis cannot express their filling. To solve 
these problems researchers had proposed emotion 
recognition methods based on Electrophysiological 
(EEG) signals and compared with other traditional 
methods [2]. 
3. Electroencephalogram (EEG) 
The cortex is divided into the frontal, temporal, 
parietal, and occipital lobes(see fig.1)[18]. The 
frontal lobe is responsible for the conscious 
thought. The temporal lobe is responsible for the 
senses of smell and sound; The parietal lobe is 
responsible for integrating sensory information 
from various senses and the manipulation of 
objects and the at last, occipital lobe is responsible 
for the sense of sight. EEG estimates voltage 
fluctuations because of ionic current flows through 
neurons of the brain. A adult EEG signal, 
measured from the scalp is of range 10-100uV. 
These signals are divided into specific ranges, 
namely the delta(1-4Hz), theta (4-7Hz), alpha(8- 
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13Hz),  beta(13-30Hz) and gamma(>30Hz) 
bands[18]. Delta waves are related with the 
unconscious brain, and occur during deep sleep. 
Theta waves are related with subconscious mind, 
and occur during sleeping and dreaming. Alpha 
waves are associated to relaxed mental state. Beta 
waves are an active state of mind. International 
10/20 system (IS) (see fig.2)[18] are used for 
standard sets of locations of electrodes on the 
skull. The number 10 and 20 shows the distance 
between neibouring electrodes. 





frontal lobe 


Fig. 1: Human brain subdivided in to temporal, 
frontal, occipital and parietal lobes. Adopted 
from [18] 





Fig. 2: International 10/20 system (IS)[18] 
The procedure of emotion recognition is as 
follows: 
e User is exposed to the stimulus are tested. 
e Changes in the voltage of human mind are 
recorded. 
e From the EEG signals artifacts are 
removed. 
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e After the removal of noise and artifact the 
signal is analyzed and the relevant features 
are extracted. 


Acquisition 
and processing 
of mult- 
channel EEG hy 

signals 





Feature 
extraction 
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e Select appropriate classifier and train them 
from the training dataset and compute 
features and predict the signal. 


Emotion 
classification 


Feature 


selection 


Fig.3: Procedure of EEG emotion recognition system 


4. Dataset and Data Preprocessing 

There are number of datasets are available 

publically that can be used for emotion 

recognition. Researchers mostly used DEAP, 

SEED and AMIGOS datasets. 

a. DEAP[39] is widely used dataset in the EEG 
based emotion recognition. There are 32 
subjects were _— participated = in the 
Electrophysiological recording. They watch 40 
one minute music videos. In the DEAP dataset 
total 32 channels were used to record. 

b. SEED[40] 15 volunteers were participated in 
which each one was watching a small clips 
having distinct emotions. In SEED dataset total 
62 channels were used for EEG recording. 

c. AMIGOS[3] 40 volunteers were participated in 
which each one was watched a set of sixteen 
short videos. Each volunteers rated each video 
on the scale of two dimensional (valence and 
arousal). 

Raw signal collected from EEG have noise and 
artifacts. Firstly we should remove this noise and 
artifacts from the raw signal. Artifacts removal is 
one of the most important step in emotion 
recognition. Artifacts are generated due to internal 
factors such as eye blinking, heart rate, muscles 
contraction and other environment factors like 
electrode position cable and recorders. There are 
many methods used for artifact removal, like linear 
regression, empirical mode decomposition, 
principal and component analysis etc. 

5. Feature Extraction and Classification 

Feature Extraction- For the better performance of 

biomedical (EEG) signals feature extractions are 

required. The main aim of feature extraction is to 
determine most important informative set of 

features to increase accuracy of the classifiers. A 

suitable feature extraction method converts one of 
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many signals into a feature vector. In the machine 
learning algorithm researchers believe that suitable 
feature extraction is the key to make an efficient 
predictive model. There are several types of 
features based on frequency domain, Time domain 
and frequency time domain for the EEG signal 
diagnosis and analysis[23]. Feature Classification- 
Machine learning is a_ subject of artificial 
intelligence. In the machine learning we train some 
network to build the predictive model. There are 
many supervised machine learning algorithm exist. 
There are various feature extraction methods are 
used by researchers to perform frequency domain, 
time domain and time frequency domain analysis 
of EEG signals. Fourier transform(FT), empirical 
mode decomposition(EMD), wavelet packet 
decomposition(WPD), wavelet transform, mRMR, 
etc are some of the feature extraction 
methods[31]. Extracted features are further 
classified using machine learning algorithms. 
Some of feature extraction techniques and 
classification techniques are discussed in this 
paper. In this section we have discuss machine 
learning techniques for emotion recognition. In[2] 
used an Ensemble Convolutional Neural Network 
(ECNN) model, which is used to automatically 
mine the correlation between multi-channel EEG 
signals and peripheral physiological signals to 
improve the emotion recognition accuracy. Author 
design five convolution networks and use global 
average pooling (GAP) layers instead of fully 
connected layers, and then the plurality voting 
strategy is adopted to establish these model. In[3] a 
deep belief-conditional random field (DBN-CRF) 
framework is used which is improved version of 
deep belief networks with glia chains (DBN-GC) 
and conditional random field. In this, feature 
vector sequence is extracted first from the EEG 
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signals. Then, parallel DBN-GC models are 
utilized to obtain the feature sequence from EEG 
signals. And then conditional random field (CRF) 
model is used to generates the emotion state 
sequence. And at the last, K-nearest neighbor 
algorithm(KNN) is used to estimate the emotion 
state. DEAP, SEED & AMIGOS dataset is used in 
this. In [4] subject independent emotion 
recognition system using Discrete Wavelet 
Transform and MLP is used. This method is better 
than the earlier methods as it has used publically 
available dataset which contains data from 32 
subjects from both the men and women. DEAP 
database is used and accuracy obtained is 58.50%. 
In [6] a feature extraction technique based on 
double tree complex wavelet transform (DTCWT) 
and machine learning algorithm is used. In this a 
Neuroscan device is used for 16 subjects with 
video stimuli. Then band pass filter is used to 
remove artifects. And at the last, support vector 
machine (SVM) is used to classify emotions: calm, 
happy, and sad. And obtained the classification 
accuracy of 90.61%. In [7] EMD method is 
utilized to decompose EEG signals with channels 
F3 and C4. A series of IMFs obtained by EMD are 
used to calculate SampEn values and to form 
feature vectors. These vectors are fed into SVM 
classifier for training and testing. The average 
accuracy of this method is 94.98% for DEAP 
dataset. In [11] a hierarchical bidirectional GRU 
model with attention mechanism (H-ATT-BGRU) 
is used for classification of emotion. The first layer 
of the model encodes the local correlation among 
the samples in an epoch, and the second layer 
encodes the temporal correlation among the EEG 
epochs in a sequence. The model uses attention 
mechanism at both sample and epoch levels. In 
[13] tunable-Q wavelet transform (TQWT) is used 
for the classification of various emotions of EEG 
signals. tunable-Q wavelet transform divides EEG 
signal into subbands and then time-domain 
features are extracted from these subbands. These 
features are used as an input to machine learning 
classifier for the classification of happy, fear, sad, 
and relax emotions. In [14] phase locking value 
(PLV) graph convolutional neural networks [P- 
GCNN] is used which is the improved model of 
PLV and GCNN. The PLV has the ability to 
segregate phase and amplitude component in the 
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EEG signal and determine inter channel 
correlation information. In [16] a hybrid feature 
extraction method is used in Empirical Mode 
Decomposition (EMD) domain with combination 
of Sequence Backward Selection (SBS) for EEG 
emotion recognition. In this detail information of 
multi scale components of EEG signal was 
extracted and optimal features was selected for 
emotion classification. Model is tested on DEAP 
dataset, in which the Valance and Arousal 
dimension emotional states are classified by K- 
nearest neighbor (KNN) and support vector 
machine (SVM). In[19] a feature extraction 
technique Teager-Kaiser Energy Operator (TKEO) 
is used along with k-nearest neighbor (KNN), 
neural network (NN) and Classification Tree (CT) 
classifiers for the emotion recognition from EEG 
signal. This study determines the performance and 
accuracy of emotion recognition which is further 
used for stress identification. In [20] dual tree 
complex wavelet transform (DT-CWT) was used 
to decomposed the EEG into five different sub 
bands from which different features were extracted 
using frequency, time and non linear analysis. In 
this deep simple recurrent units (SRU) network is 
adopted which is not only able to processing a 
sequence data but also has the ability to resolve the 
problem of long term dependencies occurs in 
normal recurrent neural network(RNN). Positive 
neutral and negative emotions were identified from 
EEG signals from emotion recognition system. In 
[23] a hybrid method of principal component 
analysis (PCA) and t-statistics is used for feature 
extraction. This spatial PCA was implemented to 
reduce signal dimensionality and select features 
based on the t-statistical. In [27] a GPSO algorithm 
for tuning the hyperparameters in the CNN model 
is used, and applies the optimized CNN model to 
the EEG emotion recognition. In [29] mRMR 
feature selection method is used after the pre- 
processing step of EEG signal to improve the 
accuracy of SVM emotion classifier on two- 
dimension(Valence and Arousal) emotions model. 
In [30] Liquid State Machines (LSM) algorithm is 
method of feature extraction and used to recognize 
the emotion state of an individual from EEG 
dataset. In this author identified arousal, valence 
and liking emotional states from the available 
dataset.[35-41]. 
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Table: 1. Various Algorithms Used in Emotional State Recognition 






























































Refer | Dataset Features Classifiers No. of Emotional State Accuracy 
ence extracted channel 
DEAP Hajorth Sadness, Anger, DEAP- 
[1 & Parameters, SVM with 32 and happiness, 84.67% , 
TYUT FD, DE,WE Gaussian Kernel 64 surprise and TYUT2.0- 
2.0 and FC neutral 89.64% 
Root mean 
square, Arousal, Valence, 
[2] DEAP Stochastic ECNN 32 liking and 82.92% 
gradient Dominance 
descent(SGD) 
Mean, Deep belief 
DEAP, | Variance, Zero conditional Arousal- 
[3] SEED & | crossing rate, random field 32 & 62 Arousal and 76.13%, 
AMIGO | approximate (DBN-CRF), Valence Valence- 
S entropy and DBN-GC and 77.02% 
PSD KNN 
Wavelet Multilayer 
[4] DEAP features (WF) Perceptron neural 1 Happy and Sad 58.50% 
network (MLP) 
Valenc- 
Wavelet Deep Neural Valence and 62.5% & 
[6] ed features (WF) network (DNN) : Arousal Arousal- 
64.25% 
NMI based Wileneeand Arousal- 
[8] DEAP channel SVM 32 73.64%, Vale 
: Arousal 
selection nce74.4% 
Statistical Pe 
fain eon Discriminative Arousal- 
(9] DEAP fein Ensemble 32 HAHV,LALV 70.1% and 
ee Learning (SDEL) HALV, LAHV Valence- 
ee with DNN and T149% 
CNN 
deep- 
Hjorth learning(bagging, 
parameters(acti | boosting, staking h 1 d 
[10] | NA __ | vity, mobility, and voting), 14 aPPY, Calm, SAC, | 76.60% 
‘ and scared) 
and naive-Bayes, 
complexity) LDA, KNN, 
SVM, 
CNN, LSTM, 
hierarchical Valence- 
PSD, SFFS and bidirectional Arousal and 69.9% and 
(1 Py AR Gated Recurrent oo Valence Arousal- 
Unit (GRU) model 65.5% 
(H-ATT-BGRU) 
12] DEAP Standard KNN, Decision 32 & 64 Negative, Calm DEAP- 
and Deviation, first tree & Random and Positive 63.09% and 
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SEED and second Forest SEED-75% 
order 
differential 
features 
DEAP PSD, phase-locking valence, arousal, Valence- 
and Differential value (PLV) graph and dominance 73.31%, 
SEED | Asymmetry(D convolutional using Arousal- 
ASM), neural networks 77.03 and 
Differential (P-GCNN) Dominance- 
[14] Entropy(DE), 32 & 64 79.20% 
Rational 
Asymmetry(R 
ASM) and 
Differential 
Caudality(DCA 
U) 
spectrum 
centroid Valence- 
[16] DEAP and Lemp eh KNN and SVM 32 Valence, Arousal BORO? 
Ziv Arousal- 
Complexity 84.90% 
(LZC) 
k-nearest neighbor 
Energy, (KNN), neural - 
[19] DEAP Feature network(NN) and 2 a _ 83.33% 
vector(FV) Classification eeelive 
Tree (CT) 
MAV, PSD, : 
fractal Decpnuuple Positive, Negative 
[20] SEED ; ; Recurrent 64 ° NA 
Dimension(FD) . and Neutral 
Unit(SRU) 
, DE 
Standard 
Deviation(SD), ANN- 
Mean Absolute 86.57% 
deviation(MA SVM- 
D), Median ANN, SVM, LDA Positive, Negative 85.85% 
[23] SEED Absolute and KNN and Neutral LDA- 
Deviation(Med 82.50% 
AD), FD, PSD KNN- 
and Spectral 73.42% 
Energy 
happy- 
sample multi-class least 92.19%, 
entropy, Tsallis fear=87.62% 
squares support 
[24] entropy, : , sad- 
: ; vector machine 
Higuchi fractal (MC-LS-SVM) 88.98%, and 
Own dimension, and happy, fear, sad, relax- 
data Hurst exponent NA and relax 93.13% 
25] Deep Neural Arousal and 
DEAP PSD network [DNN] 4 Valence 
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Statistical 
features: 
power, mean, 
Arousal- 
shai 59.77% 
deviations, 
; Valence- 
narmalized enemaoh arousal, Valence, 51.88% 
[26] DEAP difference, yP . rs a 32 Dominance and aoe 
Hiroth Partitioning Liking Liking- 
: : 62.11% 
P ene i Dominance- 
monn: 63.75% 
complexity. 
Fractal 
Dimention(FD) 
Dynamic oy 
(28) | SEED Sample SVM 62 Fos and 85.11% 
Negative 
Entropy 
Hjorth 
parameters, 
[29] Statistical Aroual- 
features and 60.7% 
Fractal Arousal and Valence- 
DEAP Dimension SVM 32 Valence 62.33% 
Decision 
Tree(DT) and 
[30] | DEAP NA Linear oy eeroMsaly Y ahetien NA 
ara and liking 
Discriminate 
Analysis(LDA) 
Own Fractal Calm, anger and 
iA dataset Dimension ova ue Happiness ee 
Mean and ee 
probabilistic HAHV,LALV 
[33] Pe Lance neural network HALV, LAHV eaeee 
(PNN) 
pees Linear Positive, 
[34] DEAP crane Discriminate 13 Negative, Angry 82% 
mean, kurtosis ; 
Analysis(LDA) and Harmony 
and skewness) 

















Conclusion and Future Direction 

In the analysis of brain signal we use EEG signals, 
as it 1s non-invasiveness high temporal resolution 
and safe nature. In this study we have surveyed the 
various methods of emotion recognition from EEG 
signals. A comparative analysis of EEG signals 
have been done in which feature extraction and 
classification methods are discussed. Our main 
focus on classification techniques and extracted 
features. From the analysis, we concluded that 
each and every stage has its own role in EEG 
signals analysis. Each and every state play a very 
important role in pre-processing EEG signals and 
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developing novel model. In pre-processing 
unwanted signals, noise and artifacts are removed 
from the raw EEG signals. In second state, feature 
extraction algorithms are used to represent high 
dimensional signal into discrete features. Then, if 
data suffers from overfitting problem, feature 
reduction techniques can be used to reduce the 
computational cost. And at last signals are 
classified with different machine learning 
algorithms and deep learning algorithms for 
emotion recognition. From the study of various 
research papers it is concluded that accuracy of 
deep learning algorithm is better than machine 
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learning algorithms. In future, we will made efforts 

in more advanced algorithms for feature 

extractions and classification techniques from the 

publically available datasets. 
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